Automatic Item Generation

The principles and practices that guide the design and development of test items are changing because our assessment practices are changing. Educational visionary Randy Bennett (2001) anticipated that computers and the Internet would become two of the most powerful forces of change in educational measurement. Bennett’s premonition was spot-on. Internet-based computerized testing has dramatically changed educational measurement because test administration procedures combined with the growing popularity of digital media and the explosion in Internet use have created the foundation for different types of tests and test items. As a result, many educational tests that were once given in a paper format are now administered by computer using the Internet. Many common and wellknown exams in the domain of certification and licensure testing can be cited as examples, including the Graduate Management Achievement Test (GMAT), the Graduate Record Exam (GRE), the Test of English as a Foreign Language (TOEFL iBT), the American Institute of Certified Public Accountants Uniform CPA examination (CBT-e), the Medical Council of Canada Qualifying Exam Part I (MCCQE I), the National Council Licensure Examination for Registered Nurses (NCLEX-RN) and the National Council Licensure Examination for Practical Nurses (NCLEX-PN). This rapid transition to computerized testing is also occurring in K–12 education. As early as 2009, Education Week’s “Technology Counts” reported that educators in more than half of the U.S. states—where 49 of the 50 states at that time had educational achievement testing—administer some form of computerized testing. The move toward Common Core State Standards will only accelerate this transition given that the two largest consortiums, PARCC and SMARTER Balance, are using technology to develop and deliver computerized tests and to design constructed-response items and performance-based tasks that will be scored using computer algorithms. Computerized testing offers many advantages to examinees and examiners compared to more traditional paper-based tests. For instance, computers support the development of technology-enhanced item types that allow examiners to use more diverse item formats and measure a broader range of knowledge and skills. Computer algorithms can also be developed so these new item types are scored automatically and with limited human intervention, thereby eliminating the need for costly and timeconsuming marking and scoring sessions. Because items are scored immediately, examinees receive instant feedback on their strengths and weaknesses. Computerized tests also permit continuous and on-demand administration, thereby allowing examinees to have more choice about where and when they write their exams. But the advent of computerized testing has also raised new challenges, particularly in the area of item development. Large numbers of items are needed to support the banks necessary for computerized 21 AUTOMATIC ITEM GENERATION